{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Autograd.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Eurus-Holmes/PyTorch-Tutorials/blob/master/Autograd.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "0mMBNmFE1DyE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# AUTOGRAD: AUTOMATIC DIFFERENTIATION\n",
        "\n",
        "Central to all neural networks in PyTorch is the `autograd` package. Let’s first briefly visit this, and we will then go to training our first neural network.\n",
        "\n",
        "The `autograd` package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, which means that your backprop is defined by how your code is run, and that every single iteration can be different.\n",
        "\n",
        "Let us see this in more simple terms with some examples."
      ]
    },
    {
      "metadata": {
        "id": "aZNNscjt1L6P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tensor\n",
        "\n",
        "`torch.Tensor` is the central class of the package. If you set its attribute `.requires_grad` as True, it starts to track all operations on it. When you finish your computation you can call `.backward()` and have all the gradients computed automatically. The gradient for this tensor will be accumulated into `.grad`attribute.\n",
        "\n",
        "To stop a tensor from tracking history, you can call `.detach()` to detach it from the computation history, and to prevent future computation from being tracked.\n",
        "\n",
        "To prevent tracking history (and using memory), you can also wrap the code block in `with torch.no_grad():`. This can be particularly helpful when evaluating a model because the model may have trainable parameters with `requires_grad=True`, but for which we don’t need the gradients.\n",
        "\n",
        "There’s one more class which is very important for autograd implementation - a `Function`.\n",
        "\n",
        "`Tensor` and `Function` are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each tensor has a `.grad_fn` attribute that references a `Function` that has created the `Tensor`(except for Tensors created by the user - their `grad_fn` is `None`).\n",
        "\n",
        "If you want to compute the derivatives, you can call `.backward()` on a Tensor. If Tensor is a scalar (i.e. it holds a one element data), you don’t need to specify any arguments to backward(), however if it has more elements, you need to specify a gradient argument that is a tensor of matching shape."
      ]
    },
    {
      "metadata": {
        "id": "In3z8R0P0dBu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "P-rbS8vJ2dpx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create a tensor and set `requires_grad=True` to track computation with it"
      ]
    },
    {
      "metadata": {
        "id": "Wuu-qZQ02VsM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "5d111a99-df2f-44ad-a0cb-70472cbd3fc0"
      },
      "cell_type": "code",
      "source": [
        "x = torch.ones(2,2,requires_grad=True)\n",
        "print(x)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1., 1.],\n",
            "        [1., 1.]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "iRLIXPkt2ocR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Do a tensor operation:"
      ]
    },
    {
      "metadata": {
        "id": "dk-MfTzS2kkP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "8a7edefa-1b17-4ead-dc55-5f284e06025b"
      },
      "cell_type": "code",
      "source": [
        "y = x+2\n",
        "print(y)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[3., 3.],\n",
            "        [3., 3.]], grad_fn=<AddBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "2TC4zq-X2xdZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "`y` was created as a result of an operation, so it has a `grad_fn`."
      ]
    },
    {
      "metadata": {
        "id": "UaSSNpSi2rUW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3402a788-6761-4a26-86c2-877b3f283bff"
      },
      "cell_type": "code",
      "source": [
        "print(y.grad_fn)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<AddBackward0 object at 0x7fd8896ecb00>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "r7tNbWoz26rn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Do more operations on `y`"
      ]
    },
    {
      "metadata": {
        "id": "WPMqXmbj22qP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "87ba8782-f2bd-42e7-a88c-b99bf33bd492"
      },
      "cell_type": "code",
      "source": [
        "z = y*y*3\n",
        "out = z.mean()\n",
        "print(z)\n",
        "print(out)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[27., 27.],\n",
            "        [27., 27.]], grad_fn=<MulBackward0>)\n",
            "tensor(27., grad_fn=<MeanBackward1>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Af9Jy3z53UEs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "`.requires_grad_( ... )` changes an existing Tensor’s `requires_grad` flag in-place. The input flag defaults to `False` if not given."
      ]
    },
    {
      "metadata": {
        "id": "izm7Ecgq3DHy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "3e13748c-1046-4ec7-e69a-1a56156e69ef"
      },
      "cell_type": "code",
      "source": [
        "a = torch.randn(2,2)\n",
        "a = ((a*3)/(a-1))\n",
        "print(a.requires_grad)\n",
        "print(a)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False\n",
            "tensor([[ 0.1686,  0.0168],\n",
            "        [-2.4499,  1.7270]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "m5C1Sosy3knl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "ad8e0448-93b9-42a4-a1ff-bc796ff88d67"
      },
      "cell_type": "code",
      "source": [
        "a.requires_grad_(True)\n",
        "print(a.requires_grad)\n",
        "print(a)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n",
            "tensor([[ 0.1686,  0.0168],\n",
            "        [-2.4499,  1.7270]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BR9PuIFF3vPi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "dd620cfc-00e1-46cd-e502-19d1c1317da6"
      },
      "cell_type": "code",
      "source": [
        "b = (a*a).sum()\n",
        "print(b.grad_fn)\n",
        "print(b)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<SumBackward0 object at 0x7fd8896eca90>\n",
            "tensor(9.0130, grad_fn=<SumBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RMfxq4ez4Nxl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Gradients\n",
        "\n",
        "Let’s backprop now. Because `out` contains a single scalar, `out.backward()` is equivalent to `out.backward(torch.tensor(1.))`."
      ]
    },
    {
      "metadata": {
        "id": "cEJ5TSf24Ae1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "423a377f-4b0c-4cf5-c73c-6344dbd8b316"
      },
      "cell_type": "code",
      "source": [
        "print(out)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(27., grad_fn=<MeanBackward1>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "IwwmJ2Ab4XGq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "out.backward()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cKlcSJAh4dxF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Print gradients d(out)/dx"
      ]
    },
    {
      "metadata": {
        "id": "bVDpf7Oi4atV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "2d7f9744-9736-49a6-9181-574ced34617a"
      },
      "cell_type": "code",
      "source": [
        "print(x)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[1., 1.],\n",
            "        [1., 1.]], requires_grad=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dzRvUZB_4gup",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "592c164d-21a6-428e-f4e3-830ea98b5b71"
      },
      "cell_type": "code",
      "source": [
        "print(x.grad)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[4.5000, 4.5000],\n",
            "        [4.5000, 4.5000]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "2ZfR8PEH5s2N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You should have got a matrix of ``4.5``. Let’s call the ``out``\n",
        "*Tensor* “$o$”.\n",
        "We have that $o = \\frac{1}{4}\\sum_i z_i$,\n",
        "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\n",
        "Therefore,\n",
        "$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n",
        "$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "0jyLTqrN5xgw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You can do many crazy things with autograd!\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "A26qMyiZ52ax",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let’s take a look at an example of vector-Jacobian product:"
      ]
    },
    {
      "metadata": {
        "id": "WQsj8loy4iZb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "895d47eb-3512-4d25-c913-0eae7448a004"
      },
      "cell_type": "code",
      "source": [
        "x = torch.randn(3,requires_grad=True)\n",
        "print(x)\n",
        "y = x*2\n",
        "print(y)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([-2.4294,  0.0200, -0.1726], requires_grad=True)\n",
            "tensor([-4.8587,  0.0400, -0.3453], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pFhfhqcQ65Tc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fb297f01-bdad-4a90-e0d4-02a3e7e42171"
      },
      "cell_type": "code",
      "source": [
        "print(y.data.norm())"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(4.8712)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "K3P2Z_2T6_2e",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ea4957f4-3b4a-4ceb-9c94-78e778e79548"
      },
      "cell_type": "code",
      "source": [
        "while y.data.norm() < 1000:\n",
        "  y = y*2\n",
        "\n",
        "print(y)"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([-1243.8374,    10.2500,   -88.3863], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "kheOicaQ8Vxy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now in this case `y` is no longer a scalar. `torch.autograd` could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to `backward` as argument:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "rU8dCyLr7KAV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7a624e86-c33e-4720-f926-f150b6219000"
      },
      "cell_type": "code",
      "source": [
        "v = torch.tensor([0.1,1.0,0.0001],dtype=torch.float)\n",
        "print(v)"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([1.0000e-01, 1.0000e+00, 1.0000e-04])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7xXSOLRU8qZ8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7f695870-519c-400d-d1d0-e84dd5019a85"
      },
      "cell_type": "code",
      "source": [
        "y.backward(v)\n",
        "print(y)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([-1243.8374,    10.2500,   -88.3863], grad_fn=<MulBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "YGtWiPwo8tXa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fa64a5e6-2ab3-4d6f-a67a-8aab491b99d3"
      },
      "cell_type": "code",
      "source": [
        "print(x.grad)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([5.1200e+01, 5.1200e+02, 5.1200e-02])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cvoL40xC9Uyz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You can also stop autograd from tracking history on Tensors with `.requires_grad=True` by wrapping the code block in `with torch.no_grad():`"
      ]
    },
    {
      "metadata": {
        "id": "7QemAV4i9Ho1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "2f4c3301-045c-4ded-92b0-180904215d37"
      },
      "cell_type": "code",
      "source": [
        "print(x.requires_grad)\n",
        "print((x**2).requires_grad)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "True\n",
            "True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "T--ho7Bd9g7X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3d82bb3d-433a-4d2a-cf6b-e624186f2782"
      },
      "cell_type": "code",
      "source": [
        "with torch.no_grad():\n",
        "  print((x**2).requires_grad)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hCbL057C9rS5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Read Later:\n",
        "\n",
        "Documentation of `autograd` and `Function` is at https://pytorch.org/docs/autograd"
      ]
    },
    {
      "metadata": {
        "id": "8B2a91O29n9D",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}